Skip to content

Anchor PS replica service lookup in the nightly CodeceptJS test - #1200

Merged
travagliad merged 3 commits into
mainfrom
claude/jolly-curie-vuohjn
Aug 18, 2026
Merged

Anchor PS replica service lookup in the nightly CodeceptJS test#1200
travagliad merged 3 commits into
mainfrom
claude/jolly-curie-vuohjn

Conversation

@travagliad

@travagliad travagliad commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Failures fixed (investigator)

  • source: nightly CI run 32140991911Nightly E2E tests Matrix (remote PMM Server), job test execution / @nightly (95723345960)
  • tests:
    • codeceptjs-e2e/tests/qa-integration/pmm_ps_replica_integration_test.js / @pmm-ps-replica-integration @not-ui-pipeline @nightly — PMM-T2029 "Verify dashboard for PS Replica Instance", data case {"serviceName":"_2"}

Our own test code, not a PMM regression. Same bug as #1178 fixed in the Playwright
suite; that PR only touches e2e_tests/**, so the CodeceptJS copy stayed broken.

What failed

Expected 4 Elements without data but found 7 on Dashboard
  …/d/mysql-replicaset-summary/mysql-replication-summary?…&var-service_name=ps_pmm_replication_8_0_1_20753…
Report Names are IO thread running,SQL thread running,Replication error no,
Replication delay,MySQL replication lag,Relay log space,Relay log written hourly

Note the URL: the data case is _2 (the replica), but the dashboard opened is
ps_pmm_replication_8_0_1_20753 — the source. Every empty panel is a
replica-status panel, empty because the source legitimately has no replica state.
The _1 case in the same run passed.

Root cause — a substring match on the node index

percona-server-setup.yml:118 appends a random suffix to both service names
(random_service_name_value: "_{{ 99999 | random + 1 }}"), so the pair is
ps_pmm_replication_8_0_1_20753 / ps_pmm_replication_8_0_2_20753.

The test selected each node with getServiceDetailsByPartialDetails({ service_name: '_2', … }),
and that helper matches with String.includes and returns the first hit:

.find((service) => Object.entries(details).every(([key, value]) => service[key].includes(value)))

'ps_pmm_replication_8_0_1_20753'.includes('_2') is true — the random suffix
_20753 starts with 2. So whenever the suffix starts with the other node's
index and that service sorts first in v1/management/services (ordered by the
random service_id UUID), the lookup silently returns the wrong instance.

That is why this is intermittent rather than permanently red: it needs both the
suffix collision (~11% of suffixes) and the unlucky ordering (~50%).

The fix

Same shape as #1178 — swap the lookup for the already-existing
getServiceDetailsByRegex with an anchored pattern
^ps_pmm_replication_.*_N(_\d+)?$, which tolerates the random suffix being
present or absent but cannot run past the node index. Two lines, no new helper.

The dropped replication_set: 'ps-async-replication' argument is redundant once
the pattern is anchored: the setups use distinct container prefixes
(ps_pmm_replication_ vs ps_pmm_gr_ / ps_pmm_ / mysql_pmm_), so
^ps_pmm_replication_ already scopes the lookup to this setup. Checked against a
shared-server inventory containing all of those — exactly one match per node.

getServiceDetailsByPartialDetails itself is left untouched; its other callers
(details_explain_test.js, the MongoDB dashboard tests) rely on substring
semantics.

Reproduction

Throwaway Linode VM, PMM Server perconalab/pmm-server:3.9.1-rc
(digest sha256:003f9c25…, the same image the failing run used), client pmm3-rc,
--database ps,SETUP_TYPE=replication. Services re-registered with the failing
run's exact names (…_20753) and re-rolled until the API returned _1 first —
i.e. the exact CI condition:

API order: ['ps_pmm_replication_8_0_1_20753', 'ps_pmm_replication_8_0_2_20753']
OLD substring .find(_2) picks: ps_pmm_replication_8_0_1_20753   <-- the source
NEW anchored regex picks:      ps_pmm_replication_8_0_2_20753   <-- the replica

Running the unmodified test against that state reproduced the failure exactly,
including the wrong service in the URL:

✔ PMM-T2029 … | {"serviceName":"_1"} in 245816ms
✖ PMM-T2029 … | {"serviceName":"_2"} in 237441ms
  Expected 4 Elements without data but found 9 …&var-service_name=ps_pmm_replication_8_0_1_20753…

(9 empty panels here vs 7 in CI — this VM is younger, so two Last 24 hours
panels are also still empty. The defect is identical: the source's dashboard
opened for the replica's assertions.)

Verification

Same VM, same colliding service names, same API ordering that produced the
failure, with the branch synced onto it:

✔ PMM-T2029 … | {"serviceName":"_1"} in 234533ms
✔ PMM-T2029 … | {"serviceName":"_2"} in 227538ms
OK  | 2 passed   // 8m

npx eslint clean on the VM (repo's own pinned config, exit 0).

Caveat: that VM run verified an earlier revision of this branch, which used a
new getServiceDetailsByRegexAndParameters helper. Simplifying to the existing
getServiceDetailsByRegex happened after teardown, so it has not been re-run on
a live server. The regex strings are byte-identical between the two revisions,
and the dropped replication_set filter was verified redundant against a
nightly-like inventory, but the end-to-end green run predates the simplification.

Not fixed here — PMM-T1142

The same nightly also failed codeceptjs-e2e/tests/QAN/timerange_test.js /
@qan — PMM-T1142, .selected-overview-row never visible after opening the
copied QAN link (job 95723345803).
Unrelated to this change and not addressed here: it passed on the previous
nightly and did not reproduce (3/3 green on the VM). For whoever picks it up —
the trace shows the selected query was present in the post-reload response and
briefly carried the highlight class before losing it, which points at QAN's own
selection state rather than a test selector. Deliberately not papered over.

getServiceDetailsByPartialDetails matches with String.includes and returns the
first hit, so the '_2' lookup in pmm_ps_replica_integration_test.js also matches
the source node whenever the setup's random service-name suffix starts with 2
(ps_pmm_replication_8_0_1_20753 contains '_2'). PMM-T2029 then opened the
source's MySQL Replication Summary dashboard while asserting the replica's
expectations, and failed with 'Expected 4 Elements without data but found 7' --
every empty panel a replica-only one.

Add getServiceDetailsByRegexAndParameters, mirroring the Playwright suite's
helper of the same name, and select both nodes with an anchored pattern. The
helper requires exactly one match rather than taking the first, so an ambiguous
pattern fails with the candidates listed instead of silently asserting against
the wrong instance.

Signed-off-by: travagliad <215686151+travagliad@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Aug 18, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 0f999822-c8ec-43c5-bffb-103d326bf69c

📥 Commits

Reviewing files that changed from the base of the PR and between 3bd7c1b and 2fbd31a.

📒 Files selected for processing (1)
  • codeceptjs-e2e/tests/qa-integration/pmm_ps_replica_integration_test.js
🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

  • percona/pmm-qa (manual)
  • percona/pmm (manual)
🚧 Files skipped from review as they are similar to previous changes (1)
  • codeceptjs-e2e/tests/qa-integration/pmm_ps_replica_integration_test.js

Included review availability: Your plan includes up to 1 review per rolling hour; 0 remain after this review.


Walkthrough

The change adds a regex- and parameter-based inventory lookup helper. Replica integration scenarios use anchored service-name matching with replication-set filtering for the current replica and the _1 replica.

Changes

Replica service lookup

Layer / File(s) Summary
Add precise inventory service lookup
codeceptjs-e2e/tests/pages/api/inventoryAPI.js
Adds getServiceDetailsByRegexAndParameters, which validates the response, filters services by name regex and exact parameters, rejects missing or ambiguous matches, and returns the unique service.
Use precise lookup in replica scenarios
codeceptjs-e2e/tests/qa-integration/pmm_ps_replica_integration_test.js
Adds anchored replica service-name matching and updates both inventory lookups to preserve replication-set filtering.
🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the anchored PS replica service lookup added to the nightly CodeceptJS test.
Description check ✅ Passed The description directly explains the intermittent replica lookup failure, root cause, fix, verification, and unrelated excluded failure.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Review feedback: the test file should not define helpers. Drop
serviceNameRegex and hardcode the anchored pattern at both call sites.
The produced patterns are unchanged.

Signed-off-by: travagliad <215686151+travagliad@users.noreply.github.com>
Review feedback: this should be as small as PR #1178. getServiceDetailsByRegex
already exists in the CodeceptJS inventoryAPI, and anchoring the pattern on
^ps_pmm_replication_ already scopes the lookup to the replication setup
(gr/single/mysql use different container prefixes), so the replication_set
filter it dropped was redundant. Revert inventoryAPI.js entirely.

Signed-off-by: travagliad <215686151+travagliad@users.noreply.github.com>
@travagliad
travagliad merged commit b96f302 into main Aug 18, 2026
26 of 33 checks passed
@travagliad
travagliad deleted the claude/jolly-curie-vuohjn branch August 18, 2026 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants